Lecture 8 – Tuesday , November 17

Let Gn : C → C denote the PGD updates at the n-sample level—that is, as applied to the objective function Ln(θ) = 1 n ∑n i=1 `i(θ;Zi). Similarly, let G denote the projected gradient operator as applied at the population level (to the function L̄ = ELn(θ)). Our goal is to bound the error ‖θ − θ‖2. Last lecture, we showed that if the population loss L̄ : C → R is (γ`, γμ) {strongly convex, smooth}, then the population operator G is contractive with parameter κ = √ 1− γ` γμ . Our goal in this lecture is to explore when the sample operator Gn inherits this type of property. In abstract terms, we can view Gn as a perturbed version of G; if the two operators are “suitably close”, then we expect that updates with Gn will yield an approximate type of contraction. The following condition formalizes this notion: